System and methods of compressed sensing as applied to computer graphics and computer imaging

ABSTRACT

Compressed sensing can be mapped to a more general set of problems in computer graphics and computer imaging. Representation of a rendered scene in the formulation y=A{circumflex over (x)} produces higher-quality rendering with less samples than previous approaches. A filter formulation Φ makes point samples compatible with wavelet and therefore allows reconstruction of 2-D images from a set of measured pixels (point samples).

This application claims the benefit of: U.S. Provisional PatentApplication No. 61/283,512 filed Dec. 4, 2009; U.S. Provisional PatentApplication No. 61/284,173 filed Dec. 13, 2009; U.S. Provisional PatentApplication No. 61/336,363 filed Jan. 21, 2010; and, U.S. ProvisionalPatent Application No. 61/392,556 filed Oct. 13, 2010.

STATEMENT CONCERNING FEDERALLY SPONSORED RESEARCH

This invention was made with U.S. government support, National ScienceFoundation grant number 0845396. The U.S. government may have certainrights in the invention.

FIELD OF THE INVENTION

The present invention relates generally to computer graphics andcomputer imaging, and in particular, the present invention relates tocompressed sensing applications to accurately estimate signals from aset of samples such as a multidimensional signal estimated from pointsamples and to accurately estimate missing pixel values to obtain anapproximation of an original signal.

The present invention may be used in a variety of applications such asmovies, video games, medical imaging, dual photography, bio-sensing,animation, computer vision such as face recognition and light transportacquisition to name a few.

BACKGROUND OF THE INVENTION

A problem in computer graphics and computer imaging is rendering, whichis the acquisition and processing of signals by a computer. A relatedproblem in imaging is known as demosaicing, which is a digital processused to reconstruct a full color image from an incomplete signal outputwith missing color information such as from an image sensor. The problemof rendering is integral to computer graphics and computer imaging thatit has been an active area of research for over 40 years.

Recently, there has been growing interest in using compressed sensingfor various applications, but not to efficiently perform computergraphics and computer imaging. For purposes of this application, theterm “image” as used herein may refer to a digital image or video streamwith more than one signal—a multidimensional signal.

Multidimensional signals are often present in computer graphics andcomputer imaging. For example, an image includes spatial dimensions suchthat it is a 2-dimensional array of numbers defining a two-dimensional(“2-D”) signal whereas a time-varying video is a three-dimensional(“3-D”) signal because in addition to the spatial dimensions, there is atime dimension. In generating graphics or images for certainapplications, the signal may have additional dimensions such as whenperforming Monte Carlo rendering. Monte Carlo rendering includescomputational algorithms that rely on a series of random samples tocompute results. For example, suppose there is a time varying scene thatis being rendered with a camera with a finite aperture. In this case,the scene might describe a multidimensional function such as afive-dimensional (“5-D”) function ƒ(x, y, u, v, t) where (x, y)describes the position of the sample on the image plane, t is the timeat which the sample is taken, and (u, v) is the position on the aperturefor depth-of-field effects. Typically Monte Carlo rendering randomlysamples the multidimensional function by picking random values for x, y,u, v, t and then evaluating the function. A 2-D image can be produced byintegrating out certain dimensions. Typically, if only a small number ofsamples are taken, the resulting image is extremely noisy andunacceptable, because there are not enough samples to properly representthe signal. As more samples are taken, the resulting image converges tothe theoretically-correct image slowly over time.

In the field of imaging, compressed sensing has been used to try toaccelerate the acquisition of images. However, most of compressedsensing techniques in this field capture an image by taking projectionsof the image with a large set of different random patterns.Unfortunately, these methods require thousands of serial measurements inorder to reconstruct a high-quality resolution image, which makes themimpractical for most real-world imaging applications, because they aresimply too slow.

In the field of rendering, it is well known that the final, renderedimage can be efficiently compressed with a transform-coding compressionalgorithm, which implies that the information content of the image canbe described more compactly than with its pixel representation.

Most systems, however, expend their effort in rendering every singlepixel in the spatial domain of the image first, only to discard theredundant information during the post-process compression. Most renderedimages are eventually compressed by some method of either static—JPEG orJPEG2000—or video—MPEG or MPEG4—image compression. Similar to capturingan image, it would be advantageous to exploit the sparsity in thetransform domain directly during rendering so that only the importantparts of the image are rendered, as opposed to rendering everythingfirst and then throwing away most of the useless information.

This seems to imply that there might be a way to accelerate therendering process directly in the transform domain. Unfortunately, theobvious approach of taking the transform of a signal is extremelydifficult for anything other than extremely simple analytic scenerepresentations. In addition, all of the known rendering algorithms todate such as ray tracing, REYES, and scanline algorithms fail to map toother transform domains.

In order to develop a framework that exploits sparsity in a transformdomain, it has to be compatible with the traditional point-samplingmethods common to all of these rendering algorithms. In the area ofimage rendering, transform compression techniques have been usedprimarily for accelerating the computation illumination. For example, anelegant hierarchical approach has been used to create a multi-resolutionmodel of the radiosity in a scene. However, this approach does notexplicitly use a wavelet basis, nor does it exploit the final imagecoherence.

Recently, interest in transform-domain techniques for illumination hasbeen renewed with research into efficient pre-computed radiance transfermethods using bases such as spherical harmonics. Again, these approachesfocus on using the sparsity of the illumination or the BidirectionalReflectance Distribution Function (“BRDF”) in a transform domain, not onexploiting the sparsity of the final image.

In terms of using transform-domain approaches to synthesize the finalimage, the most successful work has been in the field of volumerendering. In this area, both the Fourier and wavelet domains have beenleveraged to reduce rendering times. However, this approach does not mapwell to the problem of exploiting sparsity in the final image toaccelerate standard image rendering.

Most of the work in accelerating ray tracing has focused on novel datastructures for accelerating the scene traversal. However, there arealgorithms to accelerate rendering that take advantage of the spatialcorrelation of the final image, which in the end is related to thesparsity in the wavelet domain. Most common is the process of adaptivesampling in which a fraction of the samples are computed and new samplesare computed only where the difference between measured samples is largeenough such as by a measure of contrast. These adaptive sampling methodsstill compute the image in the spatial domain making it impossible toapply arbitrary wavelet transforms.

Although there are sampling/reconstruction techniques for improvingrendering to generate better quality images in shorter times than withstandard techniques, these techniques focus on sampling themultidimensional scene information efficiently instead of focusing onreconstructing the image efficiently in a transform domain.

Furthermore, many times in rendering one must compute the definiteintegral of an unknown function. Other algorithms must solve theintegral of a signal that is known but difficult to describe in analyticform suitable for integration. This problem is encountered in a varietyof application areas such as algorithms that compute the illumination ina scene, algorithms that determine the reflection off a surface with acomplex BRDF, or even algorithms that synthesize images by calculatingimage effects such as depth-of-field or motion blur for image synthesis.

The most common approach for solving these integrals in computergraphics is known as “Monte Carlo” integration, where a large number ofrandom point samples of the function are taken and used to estimate thevalue of the integral with some probability. As more samples are taken,the variance of the result is reduced and the integral is estimated moreaccurately. Theoretically, an infinite number of samples are needed toget an exact estimate of the value of the integral. In practice, it iswell known that these methods require many samples to converge, and as aresult consume most of the computation time in the algorithms.

Another problem in computer graphics occurs when generating a discreteimage of a continuous scene representation, known as antialiasing. Oneapproach specifies that a signal must be sampled at more than twice itshighest frequency, otherwise known as the Nyquist rate, in order to bereconstructed accurately. The continuous scene can have sharpboundaries, which means that theoretically the signal has infinitebandwidth and would therefore require an infinite number of samples forreconstruction. Therefore, if the signal is sampled at a rate lower thanthe Nyquist rate, aliasing will occur where high frequency content notcaptured by the sampling rate appears as lower frequency content. Inrendering, aliasing in the image appears as the “jaggies” along sharpboundaries in the scene.

Antialiasing algorithms attempt to band limit the signal so that thefrequency of the signal will be within the Nyquist rate before beingsampled. Theoretically, this can be done by the convolution with a sinc() function in the spatial domain (multiplication by a rect( ) functionin the frequency domain), but in practice other approaches are typicallyused since the sinc( ) is not a localized filter. One common approach isto use a box filter in pixel space, which effectively integrates thesignal under a pixel in order to antialias it. These integrals aretypically computed from a few fixed samples in the case of real-timerasterization hardware, or from many random samples in the case ofhigh-end rendering systems.

There is a need for a system and methods that provides an improvedapproximation of an original signal that provides better results,including the generation of graphics or images with little or nodistortion artifacts or motion blur. The present invention satisfiesthis demand.

SUMMARY OF THE INVENTION

The present invention relates to reconstructing signals of computergraphics and computer imaging applications using compressed sensing.Compressed sensing—also known as compressive sensing, compressivesampling and sparse sampling—is a technique for acquiring andreconstructing a signal utilizing the prior knowledge that it is sparseor compressible. A signal is sparse in a transform domain if it has alarge majority of zeros (0) in its transform representation.

According to the present invention, a signal may be reconstructed veryaccurately from a small set of linear measurements given that theoriginal signal is sparse in a transform domain. In computer graphicsand computer imaging, compressed sensing is used to accurately estimatesignals from a set of samples—such as a multidimensional signalestimated from point samples—and to accurately estimate missing pixelvalues.

This invention uses compressed sensing (or sparse signal reconstruction)at its core to accelerate the process of taking samples to properlyrepresent the signal by converging the resulting image to thetheoretically-correct image over time.

The present invention takes point samples, or a set of measured pixels,of a multidimensional function and reconstructs the entire signal from aset of samples assuming that the original signal is sparse in atransform domain. The present invention is applicable to 2-D signalscorresponding to images with applications in rendering and imaging. Thepresent invention is also applicable to 3-D signals corresponding tovideo streams with applications in reconstruction and motion blur. Thepresent invention is also applicable to signals in 4-D or morecorresponding to applications in global illumination rendering effects.

When dealing with 2-D signals in particular, wavelets are typically usedto achieve the appropriate amount of sparsity. However, manyapplications in both rendering and imaging require the use of pointsamples. Wavelets are fundamentally incompatible with point samples interms of compressed sensing because the wavelet basis is not incoherentwith point samples. Compressed sensing requires that the compressionbasis (in this case wavelet) be incoherent with the measurement basis(in this case point samples) because the few non-zero elements in thecompression basis are trying to be identified with a small number ofmeasurements in the measurement basis. If the two were very coherent,then taking a few measurements with one would give us very littleinformation about the other. In the case of wavelets/point-samples, itis very simple to describe a point-sample delta function with a fewwavelets. Therefore, the two are very coherent, and cannot be useddirectly in the compressed sensing formulation.

Therefore, the blurred wavelet formulation of this invention—a Gaussianfilter in the sampling step—makes wavelets compatible with pointsamples. In embodiment of the invention related to super-resolution, theGaussian filter models the downsampling process that generates the lowresolution image. Although a Gaussian filter is used, other filters arecontemplated to increase the incoherency between wavelets and pointsamples.

The present invention is applicable to problems in rendering such asglobal illumination, antialiasing, and motion blur. It is alsocontemplated that the present invention may be used in any applicationsthat use Monte Carlo to sample a multidimensional function.

The present invention can be applied to problems in imaging since pointsamples are used, and the integral projections of the other approachesis not needed. This accelerates the imaging process since a lot ofserial measurements are not needed.

The present invention can be applied to structured illumination whichcan be used for problems in graphics such as light transport acquisitionand environment matting as well as computer vision such as 3-Dreconstruction and 3-D stereo if Bernoulli patterns are used.

The present invention may also be applied to applications for imageinterpolation such as hole filling and in-painting. The presentinvention may be used for high-speed video applications. Typically, in ahigh-speed video the biggest disadvantage is bandwidth—it takes a finitetime to read all the data from the sensor. According to the presentinvention, only a fraction of the pixels at any frame needs to be read,and then a high-quality spatio-temporal volume of the samples thatweren't measured can be reconstructed. Reducing the bandwidth may permitthe increase the frame rate.

As an example, to estimate a signal x that is sparse in the transformdomain Ψ, the process can be written as y=Sx if linear measurements ofthe signal are taken, where y is the set of measurements, and S is thesampling matrix.

If the transform domain representation x=Ψ{circumflex over (x)} issubstituted into the process of linear measurements, the measurementprocess becomes y=SΨ{circumflex over (x)}=A{circumflex over (x)} where Ais the measurement equation. Compressed sensing theory allows theestimation of {circumflex over (x)} (and hence x) very accurately from asmall set of random linear measurements in y if the transform-domainsignal {circumflex over (x)} is sparse, meaning that it has a lot ofzeros thereby enabling the measurement of signals from a few samples.

According to one embodiment of the present invention, x is an image withmissing pixel information. Therefore, estimation of image x can beaccomplished by measuring a random subset of pixels using a ray tracingrendering system. Then, the missing pixels are estimated by searchingfor the sparsest {circumflex over (x)} that matches the measurements wehave made.

According to another embodiment of the present invention, compressedsensing is used to fill in the pixels of an image that has been measuredusing a Bayer mosaic. It is contemplated that this embodiment may beused in conjunction with digital cameras. Most cameras do not measureRed-Green-Blue (“RGB”) at every pixel, but rather have a specific filteron each pixel so that either red, green, or blue are measuredinterchangeably at every pixel in a specific pattern called a Bayermosaic. Specifically, a Bayer mosaic has a RGB arrangement on a squaregrid of photosensors. Although the RGB pattern is fixed, it should benoted that randomness is not essential for compressed sensing accordingto the present invention. The present invention allows the missing RGBvalues to be computed from neighboring values.

In yet another embodiment of the invention, a high resolutionreconstructed image is created from a downsampled image. A downsampledimage is an image of which the sampling rate of a signal is reduced todecrease the data rate or the size of the data. The high resolutionreconstructed image is enhanced to show details for the downsampledimage. According to this embodiment of the present invention, themissing pixel information is provided from the pixels measured at thelower resolution.

The present invention allows compressed sensing algorithms to work whenusing point samples and a wavelet compression basis. Wavelets are muchbetter at compressing images and only require approximately 3% waveletcoefficients to accurately represent an image—thus, it is approximately97% sparse. Although wavelets make the transform-domain signal{circumflex over (x)} more sparse, they are unfortunately incompatiblewith the point samples unless a filter Φ in the formulation isintroduced. Specifically, a blurred image x_(b) is assumed to exist,which can be sharpened to form the original image x=Φ⁻¹x_(b) where Φ⁻¹is a sharpening filter. The sampling process can now be written asy=Sx=SΦ⁻¹x_(b). Since the blurred image x_(b) is also sparse in thewavelet domain, the wavelet compression basis can be incorporated to gety=SΦ⁻¹Ψ{circumflex over (x)}_(b) and the standard compressed sensingframework can now be used to solve for the sparset x_(b). Once x_(b) hasbeen found, the final image can be computed by taking the inversewavelet transform and sharpening the result x=Φ⁻¹Ψ{circumflex over(x)}_(b)

With respect to one aspect of the invention, introducing a filter makeswavelets compatible with point samples in a compressed sensingformulation. In one embodiment, a Gaussian filter is used for Φ. Sincethe filtering process can be represented as multiplication in thefrequency domain, the Gaussian filter is Φ=F^(H)GF where F is theFourier transform matrix and G is the diagonal matrix with values of aGaussian function along its diagonal.

With respect to another aspect of the invention, the present inventionfacilitates reconstruction of multidimensional signals beyond 2dimensions, such as a video stream (3 dimensions), or a rendering scenewith a plurality of parameters including, for example, aperture, arealight sources, diffuse reflections, depth-of-field, sub-surfacescattering and motion blur. Extending signals to the various dimensions,often the sparsity in the Fourier domain is sufficient to reconstructthem accurately with a few samples. Therefore, the filter formulationdiscussed above is not required in this embodiment. However, the presentinvention uses compressed sensing to reconstruct multidimensionalsignals in Monte Carlo rendering.

The multidimensional function ƒ(x, y, u, v, t, . . . ) can be pointsampled and then reconstructed assuming that it is sparse in a transformdomain. If the dimensionality is high (4 or higher) Fourier transformswork well and the resulting reconstructed signal is much better thanwith conventional approaches. However, in lower dimensional cases (lessthan 4), the blur filter formulation discussed above is needed sincewavelets are used.

According to the present invention, the final image is desired inrendering applications—not the multidimensional signal. Therefore, thereconstructed signal can be integrated down over the parameters that arenot needed to produce the final image:

I(x, y)=∫∫∫ƒ(x,y,u,v,t)dudvdt. Thus, integrating over time producesmotion-blur, integrating over the (u, v) coordinate on the apertureproduces depth-of-field, and integrating over the pixel footprintproduces antialiasing. The present invention facilitates computation ofintegrals of unknown functions very efficiently.

As discussed above, point sample measurements are used, which isparticularly feasible in simulated environments (such as renderingsystems) or certain imaging systems. Yet, with respect to another aspectof the invention, other kinds of linear measurements can be performedsuch as projecting light patterns and then integrating the reflectedlight in a sensor. This allows integral projections to be performedquite easily. Therefore, the present invention provides a process ofilluminating Bernoulli random light patterns. The idea of projectingBernoulli random light patterns with a projector in order to reconstructthe signal using compressed sensing is quite useful.

According to another embodiment, the present invention is directed to aframework that exploits the known sparsity of a final rendered image ina transform domain such as wavelet, by leveraging results in the fieldof compressed sensing. Wavelet transforms are common for imagecompression. However, wavelet transforms are not the only way totransform the image; for example, Fourier, ridgelet, curvelets, andothers might also be used.

The present invention, incorporating a computer system running analgorithm, takes a small set of point samples of a captured image orscene in the spatial domain using a ray tracer and estimates the wavelettransform of the image during rendering. A ray tracer is a particularrendering system and is easy to use because of its flexibility inselecting non-uniform samples, although other rendering systems existand are also contemplated, such as a rasterizer, REYES architecture,among others.

Since the energy of the image is concentrated more compactly in thewavelet domain, less samples are required for a result of given qualitythan with conventional spatial-domain rendering.

By taking the inverse wavelet transform of the result, an accurateapproximation of the desired final image can be computed. Compressiverendering can generate visually lossless images with approximately 75%of the pixel samples using a non-adaptive scheme. Since the algorithm ofthe present invention works in image space, it is also completelyindependent of scene complexity and scales well for complex scenes.

An image is compressible in a transform domain, then optimizationalgorithms can be used to approximate the image accurately using asmaller number of spatial samples.

The present invention produces high-quality resolution images fasterthan with conventional methods. High-quality resolution images aregenerated with fewer samples. The present invention is the first toapply the theory of compressed sensing to the problem of rendering.Particularly, the present invention works with wavelet functions withinthe compressed sensing framework. Compressed sensing requiresincoherence between the sampling basis and the compression basis.Rendering systems require point samples and image compression is bestdone with wavelets, which unfortunately are not incoherent bases. Thepresent invention overcomes this challenge and renders imagesefficiently using wavelets.

The present invention synthesizes high-quality resolution images atlower sampling rates using both adaptive and non-adaptive samplingschemes. Compressive rendering achieves visual lossless-ness withouthaving to sample every pixel in the image.

The present invention integrates unknown functions, or signals, underthe framework of compressed sensing. Specifically, a theory ofcompressed sensing demonstrates that a signal from a small set of linearmeasurements can be reconstructed if the signal is sparse in a transformdomain that is incoherent with the measurement basis.

Applied to the problem of integration for antialiasing to reduce orminimize the distortion of artifacts, it is possible to compute theintegral of a sparse function exactly with a fixed set of point samples.In instances where the signal is not sparse but compressible, thepresent invention can still be used to approximate the integral of thefunction within bounds. According to the present invention, the term“compressive integration” refers to exploitation of sparsity to computethe integrals of the function they represent, or simply compressedsensing applied to integration of unknown functions.

Under the framework of compressed sensing, the integral can be exactlyevaluated when the signal is sparse in a transform domain using a fewpoint samples without requiring the lengthy iterations of existingapproaches such as the Monte Carlo approach thereby acceleratingintegration. As a result, well antialiased images are generated withfewer samples compared to traditional approaches.

In one embodiment, the present invention is a method for reconstructinga signal using compressed sensing. An original signal is provided and aplurality of random point samples of a multidimensional function isselected. The original signal is assumed to be sparse in a transformdomain. It is also contemplated that the original signal may be assumedto be sparse in the spatial domain such that no transform is necessary.The original signal is measured at one or more locations of the randompoint samples to obtain measurements. A signal is solved for in thetransform domain wherein the signal is as sparse as possible and matchesthe measurements. When solving for the signal, wavelets for compressionmay be used. Furthermore, a filter may be applied such as a Gaussianfilter. Then, measurements of missing point samples from the signal inthe transfer domain are estimated. Finally, an image is computed. It isalso contemplated that a Bernoulli pattern can be added forillumination.

According to the present invention, the original signal may betwo-dimensional, three-dimensional or multidimensional such asfour-dimensional. In one embodiment, a three-dimensional signal mayrelate to a scene moving over time, which produces an image with motionblur. In another embodiment, a two-dimensional signal may relate torendering an image and may further relate to image antialiasing. Inanother embodiment, the multidimensional signal may relate to renderingmultidimensional effects, such as depth-of-field, area light sources,diffuse reflection, sub-surface scattering. Where the multidimensionalsignal is four-dimensional, the signal may relate to light transportbetween a two-dimensional light source, such as a projector or monitor,and a two-dimensional imaging sensor.

It is an object of the present invention to accelerate the renderingprocess directly in the transform domain. It is contemplated that thepresent invention focused on reconstructing an image efficiently in atransform domain may be used with existing techniques that focus onsampling the multidimensional scene information in order to accelerateor improve rendering to generate better quality images in shorter times.

It is another object of the present invention to integrate unknownfunctions based on the theory of compressed sensing. By assuming thatthe signal is sparse in a transform domain, the framework can solve forthe most significant transform coefficients and therefore directlycompute the integral.

Another object of the present invention is to implement a newantialiasing algorithm for ray tracing that provides results superior torandom or stratified jittered sampling—both commonly used approaches inantialiasing high-end rendering.

Another object of the present invention is to synthesize an image from acaptured image or scene information. The image is synthesized by takinga small set of point samples in the spatial domain and estimating thewavelet transform of the image during rendering. A ray tracer may beused to take the point samples in the spatial domain.

Since the energy of the image is concentrated more compactly in thewavelet domain, it is an object of the present invention to use lesssamples to synthesize an image for a result of given quality than thenumber of samples used with conventional spatial-domain rendering.

It is an object of the present invention to use an adaptive compressiverendering algorithm or a non-adaptive compressive rendering algorithm tosynthesize an image from scene information.

It is an object of the present invention to generate a high-qualityresolution image from a single low-resolution input image that issuperior to that which is produced through known super-resolutionalgorithms. The high-quality resolution image is generated from a singlelow-resolution input image without any training data set. Thehigh-quality resolution image is generated by taking advantage of thecompressibility of the image in the wavelet domain to generate ahigh-resolution result. A Gaussian low-pass filter is used as a samplingmatrix. The Gaussian low-pass filter simulates the antialiasing orbandlimiting process prior to down sampling, and, as a result, duringthe inverse procedure, the image gets sharper by the inverse of theGaussian.

Another object of the present invention is to accurately fill in missingpixel information of a sensor array device that has defective pixels.Reconstruction of an image according to the present invention reducesthe bandwidth of an imaging device since only a fraction of pixels of animage is needed to generate a high-quality resolution image versus theentire array.

The described embodiments are to be considered in all respects only asillustrative and not restrictive, and the scope of the invention is notlimited to the foregoing description. Those of skill in the art willrecognize changes, substitutions and other modifications that willnonetheless come within the scope of the invention and range of theclaims.

BRIEF DESCRIPTION OF THE DRAWING

The preferred embodiments of the invention will be described inconjunction with the following drawings provided herein to illustrateand not to the limit the invention.

FIG. 1 illustrates compressive sensing reconstruction with wavelets from25% pixel samples and filtered wavelet formulation according to oneembodiment of the present invention;

FIG. 2 illustrates images reconstructed from 25% pixels usinginterpolation and compressed sensing according to the present invention;

FIG. 3 shows a comparison of stratified sampling and antialiasingaccording to the present invention;

FIG. 4 illustrates a visual comparison of motion blur results accordingto the present invention;

FIG. 5 illustrates dual photography using compressed sensing accordingto the present invention;

FIG. 6 illustrates another embodiment of dual photography usingcompressed sensing according to the present invention;

FIG. 7 illustrates reconstructed light transport of dual photographyaccording to the present invention; and

FIG. 8 illustrates global illumination effects of dual photographyaccording to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is directed to a system and methods to accuratelyestimate missing pixel values using compressed sensing or sparse signalreconstruction. According to the present invention, a signal can bereconstructed very accurately from a small set of linear measurementsgiven that the original image is sparse in a transform domain.

The present invention takes point samples, or a set of measured pixels,of a multidimensional function and reconstructs the signal from the setof samples assuming that the original signal is sparse in a transformdomain (otherwise referred to herein as compressed sensing). The presentinvention is applicable to 2-D signals corresponding to images withapplications in rendering and imaging. The present invention is alsoapplicable to 3-D signals corresponding to video streams withapplications in reconstruction and motion blur. The present invention isalso applicable to signals in 4-D or more corresponding to applicationsin global illumination rendering effects.

When dealing with 2-D signals in particular, wavelets are typically usedto achieve the appropriate amount of sparsity. However, manyapplications require the use of point samples. Wavelets arefundamentally incompatible with point samples in terms of compressedsensing. Therefore, the blurred wavelet formulation—a Gaussian filter inthe sampling step—makes wavelets compatible with point samples. Inembodiment of the invention related to super-resolution, the Gaussianfilter models the downsampling process that generates the low resolutionimage. Although a Gaussian filter is used, other filters arecontemplated to increase the incoherency between wavelets and pointsamples.

As an example, to estimate a signal x that is sparse in the transformdomain Ψ, the process can be written as y=Sx if linear measurements ofthe signal are taken, where y is the set of measurements, and S is thesampling matrix.

If the transform domain representation x=Ψ{circumflex over (x)} issubstituted into the process of linear measurements, the measurementprocess becomes y=SΨ{circumflex over (x)}=A{circumflex over (x)} where Ais the measurement equation. Compressed sensing theory allows theestimation of {circumflex over (x)} (and hence x) very accurately from asmall set of random linear measurements in y if the transform-domainsignal {circumflex over (x)} is sparse, meaning that it has a lot ofzeros thereby enabling the measurement of signals from a few samples.

According to one embodiment of the present invention, x is an image withmissing pixel information. Therefore, estimation of image x can beaccomplished by measuring a random subset of pixels using a ray tracingrendering system. Then, the missing pixels are estimated by solving forthe sparsest {circumflex over (x)} that matches the measurements made.

According to another embodiment of the present invention, compressedsensing is used to fill in the pixels of an image that has been measuredusing a Bayer mosaic. It is contemplated that this embodiment may beused in conjunction with digital cameras. Most cameras do not measureRed-Green-Blue (“RGB”) at every pixel, but rather have a specific filteron each pixel so that either red, green, or blue are measuredinterchangeably at every pixel in a specific pattern called a Bayermosaic. Specifically, a Bayer mosaic has a RGB arrangement on a squaregrid of photosensors. Although the RGB pattern is fixed, it should benoted that randomness is not essential for compressed sensing accordingto the present invention. The present invention allows the missing RGBvalues to be computed from neighboring values.

The present invention performs significantly better than bilinearinterpolation and other pixel-interpolation approaches that might beused such as in-painting, hole filling, and edge-aware interpolation, toname a few. Specifically, a wavelet basis for compression is used, whichoffers increased sparsity. It is contemplated that other bases otherthan wavelet may be used, such as ridgelets, curvelets, any othergeneral compression basis or even a patch-based dictionary. Wavelets arenot incoherent with the spatial point samples. After all, the better atransform is at defining localized features, the more coherent it willbe with the spikes of a point sample basis and the less likely it willwork with the compressed sensing framework. The present invention can beapplied to problems in imaging since point samples are used, and theintegral projections of the other approaches is not needed.

The result of trying to use a wavelet basis by itself is shown on theleft of FIG. 1. As shown on the left of FIG. 1, compressed sensingreconstruction with wavelets from 25% pixel samples has problems. Theimage on the right of FIG. 1 illustrates the result of using thefiltered wavelet formulation presented from the same samples therebyillustrating significant improvement over traditional interpolationtechniques. After the addition of the filter, the coherence between asampling basis and compression bases can be found. The filtered waveletdecreases coherence with the point-sampling basis. In one embodiment thefilter is a Gaussian filter, although any kind of filter iscontemplated. Therefore, the reduction in coherence allows use ofcompressed sensing.

According to the present invention, a simple algorithm can be performedin three steps. The first step involves the selection of random pixels.A Poisson-disk distribution—where all the pixel samples are separated byat least a fixed distance—works better than completely random samples.Once the pixels are selected, the imaging process is simulated in therendering system at these pixel locations. The missing pixels are solvedfor using a greedy algorithm. In one embodiment, the missing pixels aresolved for using Regularized Orthogonal Matching Pursuit (“ROMP”), usingDaubechies-8 (“DB-8”) wavelets for compression. It is contemplated thatany greedy algorithm may be used that approximate solutions to the l₀problem such as SpaRSA. Furthermore, it is contemplated that a l₁ solvermay be used, such as linear programming or other forms of convexoptimization. The present invention contemplates the use of othersolvers such as those developed after the filing date of the presentinvention. Once x_(b) is found, the desired image may be computed. As anexample of one embodiment, the entire reconstruction algorithm iswritten in C and takes 100 seconds on a laptop with 2.2 GHz processor toprocess a 512×512 image. A Graphics Processing Unit (“GPU”)implementation is also possible.

Unfortunately, traditional compressed sensing algorithms do not workwhen using point samples and a wavelet compression basis. Wavelets aremuch better at compressing images and only require approximately 3%wavelet coefficients to accurately represent an image—thus, it isapproximately 97% sparse. Although wavelets make the transform-domainsignal {circumflex over (x)} more sparse, they are unfortunatelyincompatible with the point samples unless a filter Φ in the formulationis introduced. Specifically, a blurred image x_(b) is assumed to exist,which can be sharpened to form the original image x=Φ⁻¹x_(b) where Φ⁻¹is a sharpening filter. With respect to one aspect of the invention,introducing a filter makes wavelets compatible with point samples in acompressed sensing formulation. In one embodiment, a Gaussian filter isused. Since the filtering process can be represented as multiplicationin the frequency domain, the Gaussian filter is =Φ=F^(H)GF where F isthe Fourier transform matrix and is the diagonal matrix with values of aGaussian function along its diagonal.

The sampling process can now be written as y=Sx=SΦ⁻¹x_(b). Since theblurred image x, is also sparse in the wavelet domain, the waveletcompression basis can be incorporated to get y=SΦ⁻¹Ψ{circumflex over(x)}_(b) and the standard compressed sensing framework can now be usedto solve for the sparset x_(b). Once x_(b) has been found, the finalimage can be computed by taking the inverse wavelet transform andsharpening the result x=Φ⁻¹Ψ{circumflex over (x)}_(b)

For comparison, the entire image from the non-uniform pixel locationsneeds to be interpolated. One traditional way to do this is totessellate the samples into a triangular mesh using Delaunaytriangulation and then bilinearly interpolate across each triangle tofill in the missing pixels. The results for the images are shown in FIG.2. Specifically, FIG. 2 illustrates images reconstructed from 25% pixelsusing interpolation and the compressive imaging framework according tothe present invention. For each inset the original, the interpolatedresult, and the result of the compressive sensing method according tothe present invention is shown.

In yet another embodiment of the invention, a high resolutionreconstructed image is created from a downsampled image. A downsampledimage is an image of which the sampling rate of a signal is reduced todecrease the data rate or the size of the data. The high resolutionreconstructed image is enhanced to show details for the downsampledimage. According to this embodiment of the present invention, themissing pixel information is provided from the pixels measured at thelower resolution.

The present invention is applicable to problems in rendering such asglobal illumination, antialiasing, and motion blur. It is alsocontemplated that the present invention may be used in any applicationsthat use Monte Carlo to sample a multidimensional function. Again, ifthe signal is sparse in a transform domain, an integral can beaccurately evaluated using a small set of point samples withoutrequiring the lengthy iterations of Monte Carlo approaches.

With respect to another aspect of the invention, the present inventionfacilitates reconstruction of multidimensional signals beyond 2dimensions, such as a video stream (3 dimensions), or a rendering scenewith a plurality of parameters including, for example, aperture, arealight sources, and motion blur. Extending signals to the variousdimensions, often the sparsity in the Fourier domain is sufficient toreconstruct them accurately with a few samples. Therefore, the filterformulation discussed above is not required in this embodiment. However,the present invention uses compressed sensing to reconstructmultidimensional signals in Monte Carlo rendering.

The multidimensional function ƒ(x, y, u, v, t, . . . ) can be pointsampled and then reconstructed assuming that it is sparse in a transformdomain. If the dimensionality is high (4 or higher) Fourier transformswork well and the resulting reconstructed signal is much better thanwith conventional approaches. However, in lower dimensional cases (lessthan 4), the blur filter formulation discussed above is needed sincewavelets are used.

According to the present invention, the final image is desired inrendering applications—not the multidimensional signal. Therefore, thereconstructed signal can be integrated down over the parameters that arenot needed to produce the final image:

I(x, y)=∫∫∫ƒ(x,y,u,v,t)dudvdt. Thus, integrating over time producesmotion-blur, integrating over the (u, v) coordinate on the apertureproduces depth-of-field, and integrating over the pixel footprintproduces antialiasing.

In one embodiment, compressed sensing is used to evaluate the integralof the continuous scene representation over the footprint of a pixel inorder to perform box-filtered antialiasing. A few random point samplesof scene per pixel are taken. These samples are positioned on anunderlying grid that matches the size of the unknown discrete functionand is aligned with its samples. A compressed sensing solver is used toapproximate a solution which can then be used to calculate the unknowndiscrete function. Once the unknown discrete function is calculated, itcan be integrated over the pixel to perform antialiasing. If the unknowndiscrete function is sparse in the transform domain, only a small set ofsamples are needed to evaluate the integral accurately.

To avoid aliasing, the size of the discrete vector must be large enoughto support twice the highest frequency of the unknown function.Normally, aliasing in image synthesis is traditionally caused byinsufficient sampling with respect to the highest frequency contained inthe signal. According to the present invention, however, the number ofsamples does not make a difference as long as it is enough to supportthe sparsity of the signal, not the highest frequency. Rather, the sizeof the problem solved using compressed sensing must be large enough toaccommodate all the frequencies.

It should be noted that the Fourier basis commonly used with compressedsensing is not suitable for antialiasing since images are actually notvery sparse in the Fourier domain. Since the quality of the algorithmaccording to the present invention depends on the sparsity of non-zerobasis function, a basis that is extremely efficient at representingnatural images is desired. Therefore, the Daubechies-8 (“DB-8”) waveletis used. Therefore, the signal is bandlimited or antialiased based onthe final image resolution. Therefore, a filter is included that issized according to the final resolution, which serves to both bandlimitthe signal based on the downsampling rate as well as reduce thecoherency between the point samples and wavelet basis. To summarize theantialiasing algorithm, random samples of the scene with a ray tracerare taken to form a measurement vector. A compressed sensing solver suchas ROMP is used to solve for the wavelet transform of the estimate ofthe high-resolution image which has been adequately filtered fordownsampling. The inverse wavelet transform is taken to compute thefiltered image in the spatial domain. Then it is downsampled with a boxfilter to compute the final image.

The addition of a blurring filter means that the measurement matrix iscomposed of two parts: the point samples and the blurred wavelet matrixwhich acts as the compression basis. This is related to reduction of theoverall coherence between the point samples and the compression basis.However, there are two subtle, but important, differences between theapproaches. First, the Gaussians are sized based on the size of thefinal image, not the sampling rate. This improves antialiasing since itenables proper filtering of the signal before downsampling. The seconddifference is that while an inverse filter can be used to get a sharp,high resolution result for direct viewing, the filtered result is usedfor downsampling because it has been appropriately bandlimited.

The results are compared against a traditional antialiasing techniqueusing random samples with Monte Carlo integration and another that usesjittered, stratified sampling to reduce variance. FIG. 3 shows acomparison of stratified sampling and antialiasing according to thepresent invention. The reference image is shown in the first column. Thereference image is rendered at 16 stratified samples per pixel. Thestratified image is shown in the second column and the antialiased imageaccording to the present invention is shown in the last column. Thestratified image of the second column and the antialiased image of thelast column are rendered at 4 stratified samples per pixel. As can beseen in FIG. 3, the images from the stratified Monte Carlo integrationshow more jaggedness.

Now the present invention is discussed in reference to the rendering ofmotion blur. Motion blur occurs in dynamic scenes when the projectedimage changes as it is integrated over the time the camera aperture isopen. Traditionally, Monte Carlo rendering systems emulate motion blurby randomly sampling rays over time and accumulating them together toestimate the integral. The approach of the present invention to motionblur is very similar to that of the antialiasing algorithm. First, a setof samples of the scene are taken, except that now the measurements arealso spaced out in time to sample the discrete spatio-temporal volume,which represents a set of video frames over the time the aperture wasopen.

Compressed sensing is used to reconstruct the representation of thevolume in transform domain. After applying the inverse transform torecover an approximation to the original set of video frames, it isintegrated over time to achieve the desired result. However, there aretwo important differences from the antialiasing algorithm. First, recallthat the reconstruction of the static image for antialiasing requiredthe addition of a filter to the wavelet basis to make it compatible withpoint sampling. Due to the existence of a volume of pixel data, thecorrelation between pixels has been increased by an extra dimension.

The sparsity of the signal in the Fourier domain is now sufficient toallow for adequate reconstruction by compressed sensing algorithms froma small set of point samples. This means that the Fourier basis for thetransform domain can be used, which is compatible with point sampling.Second, because a simple Fourier basis is present, the SpaRSA compressedsensing solver worked more robustly and faster for reconstructing thevideo data sets.

First, motion blur is tested by moving the camera as shown in FIG. 4.FIG. 4 illustrates a visual comparison of motion blur results. Thereference image shown in the first column was rendered with 70 temporalsamples per pixel, while the other two—random sampling in the secondcolumn and the algorithm according to the present invention—wererendered with a single random sample per pixel in time. As shown in FIG.4, one sample per pixel provides a reasonable motion blur, especiallywhen compared to the simple Monte Carlo approach.

The quality of the reconstructed frames of the spatio-temporal volume isactually quite high suggesting that compressed sensing could also beused to render animated scenes. The present invention uses compressedsensing to evaluate a sparse version of the signal in variousdimensions, so it fully computes the entire spatio-temporal volume whichviewed as frames in a video sequence.

The present invention can be applied to structured illumination whichcan be used for problems in graphics such as light transport acquisitionand environment matting as well as computer vision such as 3-Dreconstruction and 3-D stereo if Bernoulli patterns are used.

As discussed above, point sample measurements are used, which isparticularly feasible in simulated environments (such as renderingsystems) or certain imaging systems. Yet, with respect to another aspectof the invention, other kinds of linear measurements can be performedsuch as projecting light patterns and then integrating the reflectedlight in a sensor. This allows integral projections to be performedquite easily. Therefore, the present invention provides a process ofilluminating Bernoulli random light patterns. The idea of projectingBernoulli random light patterns with a projector in order to reconstructthe signal using compressed sensing is quite useful.

In one embodiment, compressed sensing accelerates the acquisition of thelight transport between a projector and a camera thereby enablingperformance of dual photography more efficiently. Dual photographyallows the interchangeability between a projector and a camera in ascene after the light transport between them has been measured. Thus, animage from the point-of-view of the projector as illuminated by thecamera can be computed.

To perform dual photography using compressed sensing, the measurementmatrix is chosen as a Bernoulli matrix since it is always incoherentwith a sparse inducing basis such as wavelet. Bernoulli patterns forillumination make it easy to implement the present invention because thesame simple, binary patterns can be illuminated whether or not thesignal is wavelet compressed. Thus, wavelet basis functions never haveto be illuminated.

Using a single camera-projector pair, the target resolution of the dualphotograph is set and then a bounding region in projector space thatcovers the desired scent is set in order to compute the illuminationpatterns. The size of the pixels of the Bernoulli pattern is thencomputed to the nearest pixel in projector space. After the image iscapture, the reflectance function of each pixel is independentlycomputed.

As shown in FIG. 5, the original image shown in (a) is transformed intodual images taken from the point-of-view of the projector (b). Since thelight transport between the camera and the projector has been captured,these images can be relit in both the primal domain as shown in (c) andthe dual domain shown in (d).

FIG. 6 illustrates details not easily visible in the original image. Theoriginal image is shown on the left of FIG. 6 whereas the dual image isshown on the right.

FIG. 6 illustrates how certain detail can be drastically enhanced in thedual image. By exchanging the projector and camera, an image can becomputed form the point-of-view of the projector in order to see thetext more clearly than could be seen from the camera.

FIG. 7 illustrates the comparison between an original image relit as apost-process to an original image obtained by projecting the samepattern on the projector during acquisition. The original image obtainedby projecting a pattern onto the scene is shown on the left. The imagecomputed using compressed sensing according to the present invention isshown on the right. Furthermore, global illumination effects such ascaustics and diffuse-diffuse interreflection is shown in FIG. 8. Theoriginal image is on the left of FIG. 8 and the image rendered from thelight transport acquired through the present invention is shown on theright.

By integrating all the pixels of the camera together a single-pixelcamera can be simulated that uses the projector for imaging trough dualphotography. According to the present invention, the projector bothmodulates the Bernoulli pattern and performs the imaging. Compressedsensing is used to efficiently acquire the reflectance function at afraction of the time it would take a brute-force scan.

Compressed sensing can be mapped to a more general set of problems incomputer graphics and computer imaging. Representation of a renderedscene in the formulation y=A{circumflex over (x)} produceshigher-quality rendering with less samples than previous approaches. Afilter formulation 1 makes point samples compatible with wavelet andtherefore allows reconstruction of 2-D images from a set of measuredpixels (point samples).

The described embodiments are to be considered in all respects only asillustrative and not restrictive, and the scope of the invention is notlimited to the foregoing description. Those of skill in the art willrecognize changes, substitutions and other modifications that willnonetheless come within the scope of the invention and range of theclaims.

What is claimed is:
 1. A computer system method for reconstructing asignal using compressed sensing, the method comprising the steps of:providing by a processor an original signal; selecting by the processora plurality of random point samples of a multidimensional function;assuming by the processor the original signal is sparse in a transformdomain; measuring by the processor the original signal at one or morelocations of the plurality of random point samples to obtain one or morefirst measurements; solving by the processor for a signal in thetransform domain that is as sparse as possible and matches the one ormore first measurements; estimating by the processor one or more secondmeasurements of a plurality of missing point samples from the signal inthe transform domain of said solving step; and computing by theprocessor an image.
 2. The method of claim 1 wherein said solving stepfurther comprises the step of using wavelets for compression.
 3. Themethod of claim 2 wherein said using step further comprises the step ofapplying a filter.
 4. The method of claim 3 wherein the filter is aGaussian filter.
 5. The method of claim 1 applied to rendering, whereinthe image is of a given scene representation.
 6. The method of claim 1wherein the original signal is a three-dimensional signal.
 7. The methodof claim 6 wherein the three-dimensional signal relates to a scenemoving over time, which produces an image with motion blur.
 8. Themethod of claim 1 wherein the original signal is two-dimensional signal.9. The method of claim 8 wherein the two-dimensional signal relates torendering an image.
 10. The method of claim 9 wherein thetwo-dimensional signal further relates to image antialiasing.
 11. Themethod of claim 1 wherein the original signal is a multidimensionalsignal.
 12. The method of claim 11 wherein the multidimensional signalrelates to rendering multidimensional effects including at least oneselected from the group of depth-of-field, area light sources, diffusereflection, sub-surface scattering.
 13. The method of claim 11 whereinthe multidimensional signal is a four-dimensional signal.
 14. The methodof claim 13 wherein the four-dimensional signal relates to lighttransport between a two-dimensional light source and a two-dimensionalimaging sensor.
 15. The method of claim 14 wherein the two-dimensionallight source is a projector.
 16. The method of claim 14 wherein thetwo-dimensional light source is a monitor.
 17. The method of claim 14wherein the light transport relates to dual photography.
 18. The methodof claim 1, further comprising the step of adding by the processor aBernoulli pattern for illumination.
 19. The method of claim 1, whereinsaid assuming step further comprises the step of assuming the originalsignal is sparse in the spatial domain such that no transform isnecessary.